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The statistical physics properties ol regular and irregular Sourlas codes are investigated in this 
paper by the cavity method. At finite temperatures, the free energy density of these coding systems 
. . . is derived and compared with the result obtained by the replica method. In the zero temperature 

0^ , limit, the Shannon's bound is recovered in the case of infinite-body interactions while the code rate is 

■ still finite. However, the decoding performance as obtained by the replica theory has not considered 

' the zero-temperature entropic effect. The cavity approach is able to consider the ground-state 

entropy. It leads to a set of evanescent cavity fields propagation equations which further improve 
{^JQ. the decoding performance, as confirmed by our numerical simulations on single instances. For the 

'~{ ' irregular Sourlas code, we find that it takes the trade-off between good dynamical property and high 

performance of decoding. In agreement with the results found from the algorithmic point of view, 
the decoding exhibits a first order phase transition as occurs in the regular code system with three- 
' body interactions. The cavity approach for the Sourlas code system can be extended to consider 

first-step replica-symmetry-breaking. 



PACS numbers: 02.70.-c, 89.90.+n, 89.70.-a, 05.50.-|-q 
^ ■ I. INTRODUCTION 



Efficient and reliable transmission of information in noisy environment plays a central role in modern information 
society. Error-correcting codes, as efficient encoding/decoding mechanisms, find widespread applications ranging from 
Ch ■ the satellite communication to the storage of information on hard disks. In 1948, Claude Shannon Jj proved that 
\ error-free transmission is possible as long as the code rate R (the ratio between the number of bits in the original 
(— I ■ message and the number of bits in the transmitted message) doesnot exceed the capacity of the channel (Shannon's 
Q bound). More explicitly, for the binary symmetric channel (BSC) where each transmitted bit is flipped independently 
O . with ffip rate p, the Shannon bound is expressed as i?c = 1 — ^2(p) where H2{p) = — plogjp — (1 — p) log2(l — p) is 
the binary entropy in the information theory literature 2]. This celebrated channel encoding theorem forms the core 
, of information theory. However, it doesnot tell us how to construct an optimal code that saturates Shannon's bound. 
^ • In information science many efforts have been devoted to construct (near) optimal codes 

' Based on insights gained from the study of disordered systems [3| the Sourlas code was proposed twenty years ago, 
^""^ which relates error-correcting codes to spin glass models Q. In the past decade, the statistical mechanics analysis of 
Sourlas codes has been successfully generalized to other types of error-correcting codes including low-density parity- 
^ . . check (LDPC) codes, MacKay-Neal codes. Turbo codes, etc. Methods of statistical physics, complementary to those 
*/~) ' used in information theory, enable one to attain a more complete picture of decoding process by analyzing global 
, properties of the corresponding free energy landscape. They also allow one to optimize the performances of various 
codes by changing some construction parameters. 

The procedure of constructing a Sourlas code is very simple. To infer which bit is flipped by noise at the receiving end 
of transmission, one has to introduce redundancy to the original message at the sending end. As for the Sourlas code, 
the redundancy is introduced by the Boolean sum of randomly selected message bits. Through the transformation 
= (~1)^' where xi is the Boolean bit and the Ising spin, the original bit sequence {xi} can be regarded as an 
d Ising spin configuration {^i}. In this way, the modulo 2 addition is equivalent to spin multiplication; and then the 
Sourlas code can be mapped to a many-body spin glass problem 0. In a general scenario, the original message is an 
N-dimensional vector ^ e {±1}^, M{> N) sets of interactions are constructed by taking the product of randomly 
sampled K bits from the sequence of the original message, i.e., Ja — £.ai ■ ■ ■ — li • ■ • i M). Then they are fed 

into the noisy channel. At the destination, M corrupted interactions Ja, some of which being different from those at 
the sending end, are received. The arising problem is how to infer the original bits from the knowledge of channel 
outputs, statistical properties of the channel and of the source. In the presence of weak noise, searching for the 
ground state of the corresponding spin glass model with given outputs {Ja} will lead to successful decoding. This 
decoding scheme is nothing but maximum a posterior probability (MAP) decoding. When the noise becomes strong, 
the finite temperature decoding or marginal posterior maximizer (MPM) scheme should be adopted since the ground 
state would probably contain no information about the original message 0] • 

The fully-connected Sourlas code has been studied in Ref. (H). It was shown that the Shannon's bound is achieved 
in the limit R ^ 0. Obviously, its practical potential is greatly limited. The finite rate Sourlas code, of greater 
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FIG. 1: Factor graph representation of a random construction of Sourlas codes (a) and the cavity method (b,c). (a) There 
are totally A'^ bits (circles) and M parity checks (squares) in the factor graph. Each bit (variable node) is connected to exactly 
six parity checks (function nodes), and each parity check involves three bits, (b) A single new bit i together with six parity 
checks is added to the original system denoted by the part above the dashed line, (c) A new function node a connected to 
three randomly selected bits is added to the original system. 

practical significance, has been studied later on (see, e.g., Refs. [1, It turns out that at finite coding rate 

R the Shannon's bound for the channel capacity can be attained at zero temperature at the limit of if — > oo 
[1, However, the Shannon's bound couldnot be achieved for finite K despite its practical significance. All the 
aforementioned investigations rely upon the replica method developed initially for solving the Sherrington-Kirkpatrick 
model of spin glass 0,[lH. Moreover, they are restricted to the replica symmetry (RS) assumption due to the emerging 
more complicated saddle point analysis of replica symmetry breaking. Nevertheless, recent developments in the study 
of LDPC codes showed that the one-step replica symmetry breaking (IRSB) type algorithm is able to shift the 
dynamical phase transition to a higher value as compared with RS type algorithms. Similar results were obtained 
on the finite connectivity Sourlas code system from the dynamic point of view [3, . In this work we study the 
equilibrium properties of the finite connectivity Sourlas code system by using the cavity method of statistical physics 

The cavity method has its own advantages over the replica method. The latter is based on a saddle point analysis of 
n-dimensional integral in the limit n — > 0. This analytic continuation in the number of replicas hasn't been confirmed 
to hold generally, neither has the validity of the exchange of the order of two limits (iV oo and n — > ). On 
the other hand, the cavity method adopts a direct probabilistic analysis, which makes it applicable to single problem 
instances. In this paper, it is expected that the cavity method reproduces results obtained by replica theory. Within 
the cavity framework, the entropic contribution in the zero temperature limit can be taken into account by means 
of first order corrections in temperature T, which has led to interesting insights on the ground state solution space 
properties of several disordered systems such as the random vertex cover problem and the random matching problem 
[l9t . Following the same strategy, we derive the evanescent cavity fields propagation (ECFP) equation for decoding 
Sourlas codes, and find it outperforms the traditional case where only the hard field or energetic contribution is 
considered. 

The rest of this paper is organized as follows. The model is introduced in Sec. |TT1 In Sec. IIIIl iterative equations 
for finite temperature decoding and zero temperature decoding are rederived respectively using the cavity method. 
Taking into account the entropic contribution, we also propose the ECFP equation. In Sec. llVi regular (with a single 
K value) and irregular (with several values of K) Sourlas codes are discussed. In this section, it is also observed that 
the ECFP procedure is able to improve the decoding performance by a significant amount. We conclude this paper 
in Sec. IVland make further discussions there. 
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II. MODEL 



Hereafter, we adopt the Ising spin representation of the Boolean numbers. In the Sourlas code scenario, the original 
binary message ^ G {±1}^ of length N is encoded into a transmitted binary message j" ~ { J", J2, ■ ■ ■ , JIj} of length 
M, with the a-th bit J° being the product of a subset da of the original message bits, J° = HiGaa (^^^ ^^S- E^) 
for a pictorial description, in which a parity check a is represented by a square and a message bit is represented by a 
circle). If each parity check involves K bits and each bit is constrained by C parity checks, then the coding rate is 
R = ^ = Y^. The Hammitonian of the system reads 



M 



a—l i^da 

where {ai} are referred to as dynamical spin variables for decoding and {Ja}^ is the received message. Due to the 
noise in the transmission channel, the received message may not be identical to the transmitted one { Jq}- We assume 
memoryless binary symmetric channel, i.e., 

PiJa\j"a) = pSiJa + j"a) + (1 - - J°) (2) 

where p is the flip rate. 

Introducing an inverse temperature /3 as a control parameter, the spin configuration cr is sampled with probability 

cxpi-Bnicr)) 

P{a\J) = ''^ ^ (3) 

where Z is the partition function. On the other hand, the Hamiltonian ([1]) is invariant under the gauge transformation 
ai o-i^i, J a '^a^i^Qa^i- Therefore, any general message can be mapped onto a ferromagnetic configuration 
{^i = +!}• Under this transformation, Eq. ^ can be re-written as 

P( Ja) = vKJa + !) + (!- V)KJa - 1) (4) 

In this sense, the Sourlas code is actually a multi-spin ferromagnetically biased ± J spin glass model. 

The aim of the statistical inference problem is to estimate the marginal posterior P{(Ji\J). We adopt the MPM 
estimator t,i — sgn(P(cri — 1| J) — P{ui = — 1| J)) = sgn(f7i)^. To measure the performance of decoding, one usually 

defines the overlap between the estimated bits {^i} and the original message {^i} as 

1 ^ „ 1 ^ 

1=1 i=l 

where sgn(a;) = x/\x\ for a; / 0. Evaluating {(Ji) ^ directly is computationally expensive, however, it can be well 
approximated using the cavity method presented in the next section. If we focus on typical value of the decoding 
overlap, Eq. ([5]) should be averaged over the quenched disorder, i.e.. 




where € represents the average over random constructions of codes with fixed bit's degree C. The other two types of 
quenched disorder come from the corruption process (P(J|^)) and the distribution of the original message bits P{^)- 
For simplicity, we concentrate on typical properties of the system with unbiased original message and memoryless 
binary symmetric channel. In the long message limit (N — > cxd), it is believed that the macroscopic observables for a 
given instance are independent of the particular realization of the disorder 0, . 



III. CAVITY METHOD 



Using the replica method, one is forced to work directly with the disorder average from the start, whereas the cavity 
method admits of taking the average over the quenched disorder after the computation. In this section, we derive 
the free energy at finite temperature as well as zero temperature for the finite connectivity Sourlas code system using 
the cavity method, and then extend the result to the irregular Sourlas code case. Within the cavity framework, the 
entropic contribution is considered in the zero temperature limit and the ECFP equation is proposed as well. 
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A. Finite temperature decoding 

Because of the random construction of Sourlas codes, it is reasonable to assume that the correlation between 
randomly sampled bits vanishes in the long message limit. We assume all the calculations below are within the RS 
ansatz (single-state cavity method). The results are straightforward to be generalized to IRSB case. 

As shown in Fig. [TJb), if we add one variable node to the original system, C function nodes should be added 
simultaneously. Then the partition function for the enlarged system is: 



M C 

(Ji a a^l k^da fc^l jedb\i 
" ^Bhj^bO'i 

^°'^En E n ^ ^ 

o"; 6 {aj}:jedb\i jedb\i 



2 cosh phj. 



0JbCi Uje8b\i 



(7) 



Z° 



Id 



]J[cosh/3J6(l +tanh^Ji, tanh^/ij^b)] + J|[cosh/3Jh(l - tanh/3J;, tanh/J/ij^b)] 



jedb\i 



where (Ji is the newly added spin, Z"^"^ ~ 6^P(X]a=i /^"^a OieSa '^*) ^^^^ partition function of the old system, 
hj^ii is the cavity field of variable node j when function node b is removed from the graph, j G db\i denotes the set 
of bits involved in function node b but i is excluded from this set. To derive the second equality in Eq. ([7]), we have 
made use of the absence of strong correlation between randomly chosen spins, since for one random construction of 
Sourlas codes depicted in Fig. [TJa), the typical loop size in the corresponding factor graph is of order logiV which 
diverges in ^ oo. In this sense, the joint probability of a few randomly selected spins P{crQa) is factorized as 
Pi^da) ~ Yii^da ^i'^i) where we write single node belief P{(Ji) as P{(Ji) = 2 cosh terms of the local field hi 
acting on the spin ct^. 

Upon defining the magnetization rm^b = tanh/3ft,i^b and the conjugate magnetization mb~>i = tanh (3ub—,i = 
tanh pjb Y[j£ab\i tanh/3/ij^f, where Ub^i is termed the cavity bias, one gets the free energy shift due to one variable 
node addition: 



- (3AF, = log = log JI [cosh/3Jfc(l + TJib^,)] + Yl [cosh/3Jb(l - mb^,)] (8) 

'-bedi bedi 

As the second step, one function node addition is performed (c.f. Fig. [TJc)). Likewise, the new partition function 
reads 

M 



a=l keda 



E 



a 

,ou j2 n 



(9) 



2 cosh phi^a 



= • cosh/3Ja(l + tanh pja m^^a) 

The corresponding free energy shift is —PAFa — log cosh/3Ja(l + tanh/^Ja nieSa ' 
energy density is given by 

/ = ^E^^^-^E(i^«i-i)^^« 

■i a 
\ ' / pop \ " / pop 



Finally the total free 



(10) 
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where means the average over populations of {nii^airha^i} when the population dynamics recipe [l^ is 

adopted. The second term in the final expression of Eq. PH]) can be understood as follows: When one variable node 
is added, the number of over-generated function nodes is C on average; the contribution of these nodes should 
be eliminated from the total free energy. Following the same line mentioned above, one can write function 
of {mf,^i}bgai\ai then obtain a closed set of equations in the form of distribution: 



Pim^^a) = 



jedb\i 



irib^i - t'Anhf3Jb Y\_ ^j^b 

jedb\i 



(11a) 



(lib) 



Eq. (Ilip is nothing but the belief propagation equation when applied to a single instance (one particular realization of 
Sourlas codes) Population dynamics recipe is applied to solve the recursive equations above. When the iteration 
reaches a steady state, the free energy can be computed and the marginal posterior can be well approximated by 
P((Ti) = ^+™'°'' for the sparse random graph. According to Eq. the performance of decoding is evaluated via 
m — jj ^j^j ~ J dmiP{mi)sgn-{mi), where the gauge transformation has been performed and the magnetization 
rrii obeys the distribution 



Pirn,) = 



Y]^ Q{rhb^i)dmb^i 

.bedi 



UbedtC^ + ^b^t) + UbedtC^ - ^b^z) 



(12) 



B. Zero temperature decoding 

The finite temperature decoding is facilitated through Eq. ([TT]) . However, searching for the ground state of the 
system requires performing zero temperature decoding, and the equations derived above can be further simplified. 
Taking the limit /3 oo, one obtains the recursive equations for cavity fields and biases: 



P{h^^a) 



Q{ub^i) 
and the free energy shifts 



]^ dub^iQ{ub^i) 

b^di\a 



W dhj^bP{hj^b) 

jedb\i 



6 Ub- 



5 I hi^a - ^ Ub^i 
\ bedi\a 

i - sgn{Jb Y[ ^3^b) 
jedb\i 



-AF, = C-J2 



Ub- 



bedi 



J2^b- 

bedi 



-AFa = 1 - 29 ( - Ja n ) 

\ iiEda ) 



(13a) 

(13b) 

(14a) 
(14b) 



where 9(x) is a step function taking values G(x) = for x < 0, 9(a;) = 1 for a; > 0. In Eq. (|13b[) . we take the 
convention sgn(O) = 0. Similarly, the overlap in the zero-temperature limit reads m = J dhP{h)sgii{h) where the field 
is subject to the distribution P{h) — J [Y[b(zOiQ{U'b-fi)dub^i] 5 {h — ^bedi''^b^i) where Q{ub^i) is the distribution 
of cavity biases according to Eq. (jl3bp . 



C. Evanescent cavity fields propagation 



In Sec. nil Bl only the hard field or energetic contribution is considered. We expect that the neglected entropic 
contribution will provide useful information for improving the decoding performance. To derive the ECFP equation. 
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we rewrite Eq. in terms of cavity fields: 



2h, 



h^di\a 



1 + rhb- 

1 - TOb_ 



(15) 



When we consider only the energetic contribution in the zero temperature limit, the resulting closed set of equations 
Eq. (fT3|) are called warning propagation (WP) The limit (3 ^ oo selects the ground state of the system under 
consideration, therefore WP also corresponds to the MAP estimator. However, as T goes to zero, the local field hi 
vanishes linearly in T, consequently contributes to the corresponding local magnetization p^ . That is to say, even if 
the local field takes value of zero, the non- vanishing evanescent part, defined as the coefficient of first order correction 
of cavity field with respect to T, still results in a finite magnetization. Therefore, these evanescent fields are expected 
to provide useful information for improving the decoding performance. Expanding the cavity field hi^a up to the 
first order in T, i.e., 



= 2h 



(16) 



where /i_>a is an integer corresponding to the energetic contribution and ri_»a a finite real value corresponding to the 
entropic contribution, then substituting Eq. (fTCl) into Eq. (fT^. one readily gets ECFP equations: 



Ii^a = ^ Sgn(Jb Jj-^f,) 

b^di\a j^db\i 



.a= J2 iUj-b = ovjea6V)iog 

bGdi\a 

+ I (at least one Ij^b 



n 



]edb\i 



(e'-.^o + 1) + n 



jedb\ 



(e'-.-t - 1) 



log 



0, at most {K - 2) 7,^6 = V j e db\i) 



I (/,^b ^ Vj e a6V) sgn( Jb ]^ /,^b)log(l + i?fc^,) 



(17a) 



(17b) 



where IljeabX^ = 
an event and 



n jeS6\i I Jb = JbSgn 



n kedb\i Ik->b 



sg: 



- (n; 



e96\i ^3^b 



where i?. 



exp [-sgn(/j_»b)rj^6] if |/j 



I(-) is the indicator function of 



1, and otherwise. In the 



summation of Eq. (jl7b[) . the first term corresponds to the case where Ij^b = for all j G db\i, the second term 
the case where at least one Ij^b = 0, at most {K — 2) Ij^b — and the last term the case where Ij^b ^ for all 
j G dh\i. Then the decoding can be easily performed via m = J dIidriP{Ii)Q{ri) (sgn(/i) + 1(7^ — O)sgn(ri)) where 
P, Q represent the distributions for the hard fields {li} and evanescent fields {ri] respectively when the population 
dynamics technique is used to solve the ECFP equations. Actually, in the zero temperature limit, the estimated 
message bit S,i = sgn(mi) = sgn(tanh/3/ii) = sgn(/i) if 7^ and sgn(ri) otherwise. 



D. Decoding irregular Sour las codes 

All the aforementioned computations are limited to the regular case where the check's degree K takes a single value. 
It is worthwhile to study the irregular case. The irregular Sourlas code is defined as the code with various values of 
K . We assume the check's degree K follows a distribution with two delta-peaks 

P{K) ^ -^5{K ~2) + {l --i)5{K -i) (18) 

We adopt this form of distribution for two reasons. One is the Sourlas code has perfect dynamical properties for 
K — 2 and high decoding performance for K = ?>. The other is the result can be compared with that obtained for the 
cascading Sourlas code jl^, [l^l . The formula for the total free energy density of the combined system is given by 
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FIG. 2: (Color online) The decoding performance for regular Sourlas codes with R = 0.5. The calculated mean values are shown 
and the corresponding variances are smaller than the symbol size. (a)The replica symmetry free energy density versus flip rate 
when zero temperature decoding is performed. The solid line corresponds to if = 2 case while the dotted line K — 3 case. The 
dashed line represents the case of Ti" — > oo, and the dashed-dotted line corresponds to the IRSB frozen spins solution. The arrow 
indicates the critical noise level where the Shannon's bound is achieved. (b)The replica symmetry free energy density versus 
flip rate when finite temperature decoding is performed. The decoding temperature is chosen to be Nishimori temperature 
f3p = ^ log . Insets: the overlap versus flip rate for zero temperature and finite temperature decoding respectively. 



N 



M 

pop 

C 



where K 



= {^F^)pop - ^ [7 {^FK=2)pop + 2(1 - 7) {^FK=3)pop 

7 and the code rate R — . The recursive equations are of the form 
rc-i 



PK-) 



Y[ drhbQ{rhb) 



fc=i 



Too 



^^^■b) + ]\b=l{^-^b) , 



^ P{K)K f 



A'-l 

Y[ P{mj)dr. 
i=i 



K-l 



rrib 



tanh /3 Jb JJ^ rrij 



(19) 



(20a) 
(20b) 



Eq. I20bl can be understood as follows: since 7 represents the fraction of function nodes with 2-spin interaction, for 
one randomly chosen bit, it is connected to a parity check involving two bits with probability P2 = and to that 

involving three bits with probability P3 = "^y^^-* . Obviously, P2+ P3 = 1 . The formula for zero temperature decoding 
of irregular codes can be derived similarly. In the next section, we will discuss the performance of decoding for regular 
and irregular codes respectively. 



8 



0.4 



0.3 



0.2 



0.1 



- 


' 1 






1 ' 1 ■ 
K=inf, R=0.5 

P 


- 


SG 

(m=0) 






(m=0) 








✓ 

✓ 

✓ 








✓ ~" ■ 






y 




(m=1) 






. - "i 









0.0 0.5 1.0 1.5 2.0 2.5 

T 

FIG. 3: (Color online) The phase diagram for regular Sourlas codes with K ^ oo keeping R — 0.5. The dashed line indicates 
the Nishimori line, the dotted line the boundary between spin glass (SG) phase and ferromagnetic (F) phase, the dashed-dotted 
line the boundary between F and paramagnetic (P) phase and the solid line the boundary between SG and P. 




FIG. 4: (Color online) The decoding overlap m as a function of flip rate p for regular Sourlas codes with R = 0.5. The solid 
or dashed line corresponds to zero temperature decoding for the K = 2 case while the dotted or dashed dotted line the K = 3 
case. The solid (black) or dotted (green) one represents results obtained by the conventional warning propagation (WP) while 
the dashed (red) or dashed dotted (blue) one evanescent cavity fields propagation (ECFP). The cutoff takes the value 4.0. 
Numerical simulations on on a single graph by ECFP are consistent with the mean field results. The size of the graph is 
— 10000. The decoding result on single graph is averaged over ten individual simulations for each flip rate p. The error bars 
indicate the standard deviations. Inset: a detailed view of the significant improvement using ECFP for the K = 3 case. 



IV. RESULTS AND DISCUSSIONS 



A. Regular Sourlas codes 

Properties of regular Sourlas codes have been studied using replica theory [l^ . In this section, we reproduce 
results obtained on regular Sourlas codes on the basis of the cavity method. 

For regular Sourlas codes, we consider the case oi K = 2 and K = 3 with the same code rate R ~ 0.5. In 
particular, the other cases {K > 3) show the same qualitative behavior as the K = 3 case. It is worthwhile to 
mention that Eq. (jlip yields a paramagnetic solution, i.e., P{mi—,a) — 6(rni^a),Qi'n^a^i) = S{rha^i). Following 
Eq. (jlOp . one readily acquires the paramagnetic free energy fpara = ^^(log2 + log cosh /3) and the entropy Spara = 

(log cosh /? — /3 tanh /3) +log2. In zero temperature limit, Spara = (1 — "^)log2- Since the entropy becomes negative 
when i? < 1, the paramagnetic solution is irrelevant for the error-correcting purpose. As to the spin glass phase, 
therefore, the replica symmetry should be broken, and a simple assumption (frozen spins assumption) is adopted to 
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FIG. 5: (Color online) The decoding overlap m as a function of flip rate p for regular Sourlas codes with R = 0.5. The calculated 
mean values are shown and the corresponding variances are smaller than the symbol size. The solid and dashed line correspond 
to finite temperature decoding (MPM) for the K — 2 case and K = 3 case respectively, while the dotted and dashed dotted 
line the ECFP decoding for which the cutoff takes the value 4.0. 

avoid the negative entropy's!,'!^, i.e., for low enough temperature, the system settles in a completely frozen glassy 
phase. On the transition boundary, both the frozen glassy phase and paramagnetic phase share the identical free 
energy, and the transition temperature is determined by SparaiPg) ~ 0. When T < Tg, the spin glass phase takes 
over, and the corresponding free energy density can be written as fsg — fpara{Pg), independent of the temperature. 
Besides the paramagnetic solution, there exists a ferromagnetic solution (m = 1). This solution is possible only in 
the case of if — > oo (note that R is kept finite) . The related ferromagnetic free energy with vanishing entropy could 
be derived according to Eq. (jlOp . i.e., fferro — ~ 2p), independent of the temperature as well. By identifying 

fferro with /^g. One can recover the Shannon's bound as predicted by Shannon's channel encoding theorem, implying 
Pc — 0.110028 when R = 0.5 (c.f. Fig.[2Ia), the arrow indicates this critical noise level). We report the phase diagram 
for the K ^ oo code in Fig.[3l note that the code rate is still kept to be finite. It is important to remark that when the 
finite connectivity is considered, modest loss in the final decoding quality should be paid, i.e., the decoding overlap 
will be smaller than unity. To illustrate the phase transition in the finite connectivity case, we refer to the phase with 
finite decoding overlap as the ferromagnetic phase. The transition is determined by identifying the ferromagnetic free 
energy with IRSB frozen spins free energy, then the glassy phase (m = 0) sets in to replace the ferromagnetic phase 
(finite m). The corresponding critical noise level is obviously smaller than the point where the magnetization (more 
precisely the decoding overlap) drops to zero. 

To solve Eq. (|lip . population dynamics technique introduced in Ref. Tf\ is applied. The size of population is taken 
to be of order 10*. Results are reported in Fig. [2l For i^T = 2, no prior knowledge of the original message is required 
for decoding, and the phase transition is of second order. As shown in Fig.l^^a), the critical noise level is determined 
by the point where the IRSB frozen spins free energy coincides with the RS free energy. After the transition, the spin 
glass phase dominates and the corresponding free energy is fixed to be fsg. Conversely, the phase transition is of first 
order for K = 3, and there is a remarkable drop in the free energy profile. However, the computed free energy, which 
seems to be lower than the frozen spins one, is unphysical after the phase transition because of its corresponding 
negative entropy. Therefore the RS assumption is incorrect and many states assumption should be adopted. The 
performance of finite temperature decoding is also shown in Fig. [2ljb). The decoding temperature is chosen to be the 
optimal one, Pp = i log named Nishimori temperature [l,l3|- In this case, the thermal temperature is identical 
to the noise temperature, and it is observed that the performance is better than that of zero-temperature decoding. 
Actually, the average spin alignment m of decoding at Nishimori temperature sets an upper bound for all achievable 
alignments 0. As our numerical simulation has shown, only the Nishimori temperature survives to get high overlap 
when the critical noise level is approached. In contrast to the K ^ 2 case, the case oi K = 3 improves the decoding 
performance significantly. However, the basin of attraction (BOA) shrinks dramatically. We have to assume initial 
bias TO/ — 0.8 for finite temperature decoding and to/ — 0.75 for zero temperature decoding. The compromise between 
good dynamical properties on one side {K — 2) and good performance on the other side {K — 3) triggered us to 
investigate the properties of the combined system with various K . 

To further improve the decoding performance in the limit when the temperature goes to zero, we have proposed 
the ECFP equation in Sec. IIII CI The decoding overlap is plotted against the flip rate in Fig. ID Results obtained 
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FIG. 6: The decoding performance for irregular Sourlas codes with R — 0.5. The calculated mean values are shown and the 
corresponding variances are smaller than the symbol size. The solid line corresponds to zero temperature decoding while the 
dotted line finite temperature decoding. Inset: the overlap versus flip rate. 

by WP are also shown for comparison. Apparently, the decoding performance is improved within an intermediate 
range of flip rate. In the presence of weak noise, most of the propagating cavity fields take values larger than 2 and 
the energetic contribution plays a dominant role. Thus both methods lead to identical performance. Once the noise 
becomes no more small, the decoding performance achieved by ECFP starts to deteriorate due to the divergence of 
some of the evanescent fields. If we set a cutoff (e.g., 4.0), to our surprise, the problem mentioned above can be 
successfully circumvented. As shown in Fig. [4l the result indeed outperforms that obtained by WP which neglects 
the entropic effects. This can be understood as follows, as flip rate becomes high enough, the relevant cavity fields 
with |/i_>a| = 1 or Ii—>a = 0, emerge and contribute to the entropic effects [l9|. These information, omitted by 
WP, is correctly extracted by ECFP procedure, and the decoding performance is finally boosted. According to our 
numerical simulations, the value (e.g., 3.0, 4.0, 5.0) we choose for the cutoff doesnot affect the decoding results. When 
zero temperature decoding is concerned, we have observed that ECFP is able to do a better job than WP since the 
entropic effects have been incorporated. However, its decoding performance still lies beneath that achieved by the 
optimal decoding (MPM) where the decoding temperature is chosen to be the Nishimori type. However, for MPM, one 
has to have a prior knowledge of the channel noise, i.e., the flip rate of the noise. We present the comparison between 
these two different kinds of decoding in Fig. \5\ In order to validate the mean field results, we run the ECFP decoding 
algorithm on a single instance. The comparison is shown in Fig. [4l The size of the graph is set to be = 10000 and 
the code rate R = 0.5. For one iteration step, messages from each bit on the graph are updated one time on average. 
We also set the maximal number of iteration steps T to be 500. The decoding result on single graph is averaged over 
ten individual simulations for each flip rate p. As observed in our simulations, the number of iteration steps, around 
p = 0.12, exceeds the preset value on most of the presented instances, which manifests the ECFP starts to lose the 
convergence on a single graph. However, the agreement with the mean field result is indeed remarkable. 



B. Irregular Sourlas codes 

As defined above, the irregular Sourlas code is a combined system with various values of K. It takes well the 
trade-off between excellent convergence property of \ow-K codes and high decoding performance of high-if codes. 
From the algorithmic point of view, the irregular code is also termed cascading code put forward in Ref. (20| and 
further studied in Ref. [isj . In this section, we report results on typical properties of the combined system based on 
the cavity analysis presented in Sec. IIII Dl 

To retain the same code rate R = 0.5, we choose C = 5,7 = 0.5 as code construction parameters. Population 
dynamics recipe is used to solve Eq. (I^O)) . and the size of population is assumed to be of order lO''. As shown in 
Fig. [6l t he combined system exhibits a first order phase transition as the K = ^ case, which was also observed in 
Ref. [lal where the cascaded encoding/decoding scheme was employed. After this transition, the free energy crosses 
over to a lower value. However, as the flip rate p increases to a high enough value, the RS entropy will be negative 
and the RS assumption is then incorrect, indicating replica symmetry should be broken. As observed in Fig. [SI the 
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finite temperature (Nishimori's temperature) decoding is superior to the zero temperature one when the noise level 
becomes no longer low. Compared with the regular code of = 3, the BOA for the combined system becomes larger 
thus we only need to take the initial bias m/ = 0.6. Additionally, the overlap of decoding for the combined system 
is higher than that of K — 2. Therefore, results demonstrated in Fig. [5] provide us an opportunity to construct an 
optimal code. As has been stated in Refs. [IMUfllj one can use multiple values of K in the interactions. As a first 
step, belief propagation or IRSB algorithm is run on a partial system with only low K [K ~ 2) interactions since the 
low- if code has perfect convergence properties. The end overlap at the first stage is expected to be well within the 
BOA of the combined system. Once higher body (e.g., if = 3) interactions are invoked, an end overlap higher than 
the one obtained by the initial step will be resulted in. 

V. CONCLUSIONS AND FUTURE PERSPECTIVES 

In this work, we have studied the finite connectivity Sourlas code based on the cavity method. Conventional replica 
results on the regular code are cross-checked. Moreover, this cavity analysis is extended to the irregular case. Typical 
properties of the combined system are investigated. It is shown that the decoding for the combined system exhibits a 
first order phase transition as occurs in the regular case {K = 3). The combined system is of two striking features, one 
is the initial bias required for convergence is degraded, the other is the final performance is enhanced. Actually, this 
does mean that the good dynamical properties (large BOA) and high decoding performance should be compromised 
in the algorithmic implementation. Thus introducing gradually higher K interactions seems to be an effective way to 
take advantage of this trade-off. 

As for the regular codes system, the evanescent cavity fields propagation equation is proposed for the first time. And 
it is capable of extracting the entropic information in the zero temperature limit, thus the decoding performance is 
considerably enhanced compared with the traditional case where only the hard field is taken into account. Numerical 
simulations on single instances are compatible with the mean field results. 

The cavity methodology, applied in our work, is very promising. Unlike replica trick, it formulates assumptions in 
a more explicit manner, even opens the way to algorithmic implementations on one single instance. In this work, we 
also discovered that the system shows negative entropy in the presence of low enough decoding temperature and high 
enough flip rate, therefore IRSB is needed for further investigation on the finite connectivity Sourlas code. Fortunately, 
the cavity method can be easily generalized to IRSB case. Meanwhile, the IRSB frozen spin glass scheme we have 
adopted in Sec. IIV Al could be also cross-checked. On the other hand, further study is required for the combined 
system to elucidate under what conditions the channel capacity is achieved |21| . Finally, the methodology is expected 
to be applied to more practical codes like LDPC codes. These lines of research are currently under way and these 
further investigations are anticipated to provide deeper insights into a variety of codes with low density nature of 
constructions. 
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